Method for accelerating the joining of distributed datasets by a given criterion

Introduction: rapidly growing volumes of information pose new challenges to modern data analysis technologies. Currently, based on cost and performance considerations, processing is usually performed in cluster systems. One the most common related operations analytics joins datasets. Join an extremely expensive operation that difficult scale increase efficiency distributed databases or systems MapReduce paradigm. Despite fact a lot effort has been put into improving this operation, often proposed methods either require fundamental changes structure, are aimed at reducing overhead such as balancing load network. Objective: develop algorithm accelerate integration sets Results: review Apache Spark architecture features computing performed, typical for combining datasets analyzed, main recommendations optimizing presented, allows you speed up special case implemented presented. This uses partitioning partial transfer nodes cluster, way take advantage merge broadcast associations. The experimental presented demonstrate method all more effective larger volume input data. So, 2Tb compressed data, acceleration ~37% was obtained comparison with standard SQL.